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Background 


•  Department  of  Defense  (DoD)  obligated  over  $300B 
in  FY2013  contracts  (GAO,  2014) 


•  Deficiencies  in  services  acquisition  has  undermined 
DoD’s  ability  to  obtain  best  value 


•  USD(AT&L)  has  called  for  improving  tradecraft  in 
services  contracting  by  strengthening  the  contracting 
process 


•  The  use  of  contractor  past  performance  information  is 
a  critical  part  of  the  services  contracting  process. 
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Background 


•  GAO  has  identified  process  deficiencies  in  DoD 
documentation  and  management  of  CPARS  reports 

-  Not  always  completed 

-  Reports  submitted  late 


-  Report  narratives  are  insufficiently  detailed  or 
are  in  conflict  with  associated  objective  scores 


•  CPARS  deficiencies  provide  less-than-optimal 
information  to  the  acquisition  team  that  relies  on  these 
reports  for  source  selection  purposes. 
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Past  Research 


•  Purpose  of  our  research  was  to  use  CPAR  data 
to  identify  contract  variables  that  drive  the 
success  of  services  acquisition. 

•  Research  Question:  Do  the  following  variables 
affect  the  success  of  a  service  contract? 

-  Service  Type 

-  Contract  Amount 

-  Level  of  Competition 

-  Contract  Type 
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Contract  Variable 


Type  of  Service 


Contract  Amount 


Level  of 
Competition 


Contract  Type 


Past  Research  Design 


CPAR  Area 
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CPARS 


Contractor  Performance  Assessment  Reporting  System 


Assessment  Areas 

Quality 

Schedule 

Cost  Control 

Business  Relations 

Management  of  Key  Personnel 

Utilization  of  Small  Business 


Obiective  Ratings 

Exceptional 

(5) 

Very  Good 

(4) 

Satisfactory 

(3) 

Marginal 

(2) 

Unsatisfactory 

(1) 

A  contract  for  which  the  contractor  received  either 
Marginal  or  Unsatisfactory  rating  in  any  assessment  area 
was  considered  as  an  unsuccessful  contract. 
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Empirical  Study 


Total  Contracts 

Total  Army  MICC  Non- 
System  Contracts 

14,395 

Less:  Non-R,  J,  S,  D 

Service  Contracts 

8,774 

Total  R,  J,  S,  D  Service 
Contracts 

5,621 

Less:  R,  J,  S,  D  Service 
Contracts  at  other  MICC 

4,906 

R,  J,  S,  D  Service 

Contracts  at  MICC  FDO 
Eustis,  Knox,  Hood, 

Bragg,  Sam  Houston 

715 

Fort  Eustis 

Fort  Knox 

Fort  Hood 

Fort  Bragg 

Fort  Sam  Houston 

238 

119 

114 

55 

189 
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Failures 

Success 

Total 

Failure  Rate 

Contracts 

22 

693 

715 

3.08% 

Successes 
Average  Ratings 

Failures  Average 
Ratings 

Quality 

4.19 

2.5 

Schedule 

4.19 

2.5 

Cost  Control 

4.1 

2.31 

Business  Relations 

4.17 

3 

Management  of  Key  Personnel 

4.18 

2.68 

Utilization  of  Small  Business 

4.07 

2.5 
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Empirical  Study 

^ - 

Finding:  The  S  type  services  (Utilities  and  Housekeeping) 
had  the  highest  failure  rate  of  all  the  product  service  codes 
analyzed.  (p- value  =  0.761;  Not  significant) 
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Empirical  Study 


Finding:  Contracts  awarded  competitively  had  the  highest 
failure  rate  when  compared  to  the  other  contracts. 

(/?-value  =  1.00;  Not  significant) 
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Empirical  Study 


Finding:  Contracts  structured  as  a  combination  contract  had 
the  highest  failure  rate  when  compared  to  the  other  five  types 
of  available  contracts. 
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Empirical  Study 


Finding:  As  the  percentage  of  1102  filled  billets  increased,  the 
contract  failure  rate  decreased. 


iiiiii 


6.00% 

5.00% 

*  4.00% 
a 

4-» 

fO 

a!  3.00% 

'ro 

2.00% 

1.00%  H 

0.00% 


4.55% 


4.94% 


4.10% 


2.15% 


1.01% 


50-60%  61-70%  71-80%  81-89%  90-100% 

Percentage  of  Authorized  1102  Billets  Filled 


13 


esSL 

.'I.*,.  !..iini 


NAVAL 

POSTGRADUATE 

SCHOOL 


Current  Research  Objective 


•  Purpose  of  Research:  Determine  the  value  of 
the  CPARS  narratives  in  services  acquisition  by 
comparing  the  relationships  between  the 
subjective  narratives  and  the  objective  scores. 
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Research  Questions 


1 .  What  is  the  added  value  of  the  contractor  performance 
narratives  beyond  the  value  of  the  objective  scores  for 
performance? 

2.  What  is  the  statistical  relationship  between  the 
sentiment  contained  in  the  narratives  and  the  obj  ective 
scores  for  contractor  evaluations? 

3 .  To  what  degree  do  the  interview  findings  contradict, 
support,  or  enhance  the  findings  for  our  research 
questions? 
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Research  Design 


•  Sentiment  analysis 

-  The  narratives  in  7 1 5  Army  services  contract 
CPARS  reports  were  reviewed 

-  Student  researchers  scored  each  narrative  along  the 
dimensions  CPARS  Quality  Checklist  (Quality, 
Robustness,  Compliance,  Value,  and  Content)  and 
compared  those  scores  with  the  objective  rating 
scores  from  the  CPARS  report  cards 

-  Independent  researchers’  scores  were  compared 
across  a  small  sample  to  ensure  inter-rater 
reliability 
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Research  Design 


•  Statistical  analysis 

-  Relationship  between  sentiment  analysis  scores 
and  associated  objective  rating  scores 

-  Investigate  the  extent  of  the  value  of  the 
narratives 

•  Interviews 

-  Conducted  with  DoD  contracting  officers 

-  Focused  on  agencies’  use  of  CPARS  and  value 
of  narratives  compared  to  obj  ective  ratings 
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Statistical  Analysis  Results 


Result 

Overall 

Unsuccessful 

Contract 

Successful 

Contract 

Successful  vs 

Unsuccessful 

Statistically 

Significant? 

P  Value 

Narrative 

Addresses  All 

Performance 

Areas  Assessed? 

No 

18.46% 

4.55% 

18.90% 

Yes 

0.043 

Yes 

81.54% 

95.45% 

81.10% 

Narrative  Based 
On  Objective 
Data? 

No 

22.52% 

0.00% 

23.23% 

Yes 

<.01 

Yes 

77.48% 

100.00% 

76.77% 

Narrative  Is  Free 

Of  Statements  To 

Avoid? 

No 

3.08% 

13.64% 

2.74% 

Yes 

<.01 

Yes 

96.92% 

86.36% 

97.26% 
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Statistical  Analysis  Results 


Result 

Overall 

Unsuccessful 

Contract 

Successful 

Contract 

Successful  vs 

Unsuccessful 

Statistically 

Significant? 

P  Value 

Narrative  Is 

Robust  & 
Comprehensive? 

No 

37.48% 

9.09% 

38.38% 

Yes 

<.01 

Yes 

62.52% 

90.91% 

61.62% 

Could  A  Layman 

Understand  Work 

Performed? 

No 

35.52% 

18.18% 

36.08% 

Yes 

0.042 

Yes 

64.48% 

81.82% 

63.92% 
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Statistical  Analysis  Results 


Result 

Overall 

Unsuccessful 

Contract 

Successful 

Contract 

Successful  vs 

Unsuccessful 

Statistically 

Significant? 

P  Value 

Is  The  Narrative 

Beneficial  Above 
&  Beyond 
Objective  Scores? 

Score  1 

11.61% 

0.00% 

12.12% 

Yes 

<.01 

Score  2 

21.68% 

0.00% 

22.37% 

Score  3 

27.97% 

18.18% 

28.28% 

Score  4 

20.84% 

27.27% 

20.63% 

Score  5 

17.76% 

54.55% 

16.59% 

Does  The 

Narrative 

Correlate  To 

Objective  Scores 
Assigned? 

Score  1 

1.82% 

4.55% 

1.88% 

No 

0.141 

Score  2 

6.43% 

4.55% 

6.49% 

Score  3 

27.83% 

9.09% 

28.43% 

Score  4 

40.42% 

40.91% 

40.40% 

Score  5 

23.36% 

40.91% 

22.80% 
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Findings  from  Statistical  Analysis 


•  In  comparison  with  successful  contracts,  the 
unsuccessful  contracts  tend  to  have  narratives  that 

-  Address  all  performance  areas  more  frequently, 

-  Are  based  on  objective  data, 

-  Are  more  robust,  and  comprehensive, 

-  Can  help  a  layman  understand  the  work  performed  more 
frequently 

-  Provide  very  good  or  exceptional  amount  of  beneficial  data 
more  frequently 

-  Are  exceptionally  successful  more  often  in  describing 
accurately  why  the  objective  scores  are  assigned  as  they  are 

21 
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Findings  from  Interviews 


•  The  appropriate  amount  of  weight  that  should  be 
assigned  to  the  past  performance  source  selection 
criteria  should  be  correlated  to  the  source,  availability, 
quality,  and  relevancy  of  the  past  performance 
information 

•  CPARS  data  is  still  often  not  reliable,  robust,  or 
comprehensive  enough.  This  results  in  source  selection 
officials  not  placing  a  significant  amount  of  weight  on 
the  past  performance  evaluation  criteria. 
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Findings  from  Interviews 


•  The  appropriate  amount  of  weight  that  should  be 
assigned  to  the  past  performance  source  selection 
criteria  should  be  correlated  to  the  source,  availability, 
quality,  and  relevancy  of  the  past  performance 
information. 

•  When  the  objective  scores  and  narrative  sentiment  are 
mismatched,  contracting  professionals  tend  to  give 
more  weight  to  the  narrative  versus  the  objective 
scores. 


WWW.NPS.EDU 


23 


NAVAL 

POSTGRADUATE 

SCHOOL 


Findings  from  Interviews 


•  Contracting  officers  are  not  always  applying  due 
diligence  in  identifying  the  appropriate  contractor  entity 
in  the  CPARS  reports.  This  is  resulting  in  contractor 
past  performance  information  not  being  fully  accessible 
in  PPIRS. 

•There  is  a  lack  of  reliable,  robust,  and  comprehensive 
amount  of  past  performance  information  available  in 
PPIRS.  This  results  in  source  selection  officials 
soliciting  contractors  for  references  or  asking 
contractors  to  fill  out  a  past  performance  questionnaire. 
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1 .  Training  should  focus  on  developing  comprehensive 
narratives  ensuring  that  acquisition  team  members  can 
fully  understand  the  work  performed,  address  all 
performance  areas  assessed  in  their  objective  scores,  and 
ensure  the  narratives  are  based  on  objective  data. 

2.  Improving  the  quality  of  past  performance  report 
submissions  will  allow  acquisition  teams  to  assign 
higher  weights  to  past  performance  evaluation  criteria  in 
source  selection  decisions. 
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Recommendations 


3.  Additional  data  analysis  tools  should  be  incorporated 
into  the  PPIRS  database  to  better  assist  contracting 
professionals  in  identifying  past  performance  trends  for 
a  particular  contractor  or  specific  type  of  service. 

4.  COR  manning  levels  should  be  reviewed  throughout 
the  DoD  to  ensure  that  organizations  have  sufficiently 
filled  billets  to  manage  the  CPARS  process. 
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Empirical  Study 


Finding  2:  Contracts  with  a  dollar  value  from  $50  million  to 
$1  billion  had  the  highest  failure  rate  of  all  the  contract 
categories.  (p-v alue  =  0.036;  Significant) 
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Empirical  Study 


Finding  5:  Regional  MICC  offices  that  had  spent  between  $0 
and  $500  million  in  annual  workload  had  the  highest  failure 
rate. 
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Empirical  Study 


Finding  6:  MICCs  that  completed  3,501  to  7,000  contract 
actions  annually  had  the  highest  failure  rate  when  compared  to 
MICCs  that  completed  3,500  or  fewer  contract  actions. 
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Empirical  Study 


Finding  7:  The  category  with  0  to  50  1102  billets  had  the 
highest  failure  rate. 
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Recommendations 


1 .  Place  additional  emphasis  on  procurement  planning,  source 
selection,  and  contract  administration  of  S  type  services 
(Utilities  and  Housekeeping). 

2.  Provide  additional  review  of  higher  value  contracts  in  the 
areas  of  developing  PWSs,  cost  estimates,  budgets,  and 
service  delivery  schedules. 

3.  Provide  additional  training  on  proposal  evaluation, 
specifically  evaluating  cost  and  technical  proposals,  as  well 
as  project  schedules. 

4.  Provide  additional  emphasis  on  procurement  planning, 
source  selection,  and  contract  administration  of  combined 
cost  type  and  fixed  price  services  contracts. 

5.  Ensure  sufficient  1102  billets  for  each  organization  and  that 

these  billets  are  filled  with  trained  contracting 
professionals.  34 
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